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Abstract 

Though demonstrated as an effective strategy for enhancing academic performance and course persistence in 
higher education, traditional Supplemental Instruction (SI) relies on face-to-face interaction in a classroom 
setting. Consequently, students who have other obligations or feel apprehensive in a group setting often 
cannot attend traditional SI sessions. This paper focuses on an innovative alternative to traditional SI: an 
online SI program currently being implemented at Texas A&M University-Corpus Christi (TAMUCC). This 
paper describes TAMUCC's online SI program and discusses results from a pilot study that compared the 


STEM course performance and persistence of TAMUCC undergraduates (N=585) randomly assigned to SI 


groups (i.e., traditional SI or online SJ) in the spring semester of 2015. 
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Undergraduate STEM Courses 

Due an increased emphasis on course persistence and institutional retention in higher education, and 
especially in STEM departments, many institutions are experimenting with ways to increase positive student 
outcomes through academic support programs. Among these programs, Supplemental Instruction (SI) 
continues to be one of the most popular. First developed at the University of Missouri-Kansas City in 1973, SI 
is a collaborative, peer-assisted model of academic assistance that employs regularly scheduled, out-of-class 
study sessions (Arendale, 1993). SI is non-remedial, targeting traditionally difficult courses with high rates of 
student attrition and/or failure rather than individual students. It is free and open to all students in these 
traditionally difficult classes but is not mandatory. SI sessions give students an opportunity to discuss course 
material and learning strategies and are facilitated by an SI Leader, a student who has previously taken and 
excelled in the course. The point of SI is not to re-teach or introduce new material. Instead, the overall goal is 
to combine what students need to learn with how they can most effectively learn it in an open, comfortable 
environment. 

The evidence supporting the effectiveness of traditional SI, which occurs face-to-face and in-person, is 
considerable (Bowles, McCoy, & Bates, 2008; Dawson, van der Meer, Skalicky, & Cowley, 2014; Malm, 
Bryngfors, & Marner, 2012; Price, Lumpkin , & Seemann, 2012; Rath, Peterfreund, Bayliss, Runquist, & 
Simonis, 2011). However, in spite of SI's popularity and effectiveness, it does not always fit the needs of all 
students. For example, nontraditional students, whose enrollment numbers at institutions of higher education 
have been growing steadily ' over the past four decades , often have financial, family, and other obligations that 
make it difficult for them attend face-to-face SI sessions held in campus classrooms during customary daytime 


instructional hours. Other students struggle in traditional SI sessions because they feel socially inhibited — or 


iIn the fall of 1970, 27.8% of persons enrolled in degree-granting postsecondary institutions were at least 25 years old. By the fall of 
2012, persons age 25 and older represented 40.5% of enrolled students at such institutions (U.S. Department of Education, 2014). 
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unable to keep pace with the Leader. However, traditional SI's limitations are not restricted to the challenges it 
presents to students, as the implementation of traditional SI often forces institutions to overcome significant 
logistical problems such as where and when to hold sessions when available classroom space is in short 
supply. 

To address the aforementioned challenges, TAMUCC began exploring alternatives to traditional SI and 
ultimately decided to investigate the feasibility and efficacy of utilizing online SI with STEM courses that have 
been historically difficult (meaning that typically 30% or more of enrolled students earn a grade of D or F or 
withdraw prior to the end of the semester) for the institution's undergraduates. To fund this investigation, 
TAMUCC applied for and was awarded, in October 2014, a First in the World (FITW)* grant (P116F140206) 
from the U.S. Department of Education. 

Online SI is essentially the same as traditional SI, except that SI Leaders and participants interact 
through a personal computer or other hardware device instead of in a face-to-face environment (Boggs, 
Heaney, Kramer, & Williams, 2011). SI Leaders ask questions and share content such as study guides, 
exercises, videos, PowerPoint presentations, and other documents on the virtual whiteboard. SI Leaders and 
participants communicate with one another by using a microphone and headset or by typing, which allows 
participants to receive feedback and communicate with the SI Leader without being constrained to a particular 
location. Moreover, because online SI sessions are recorded, students can view them anytime and as many 
times as they wish. 

Although a relatively recent phenomenon and not nearly as well studied as traditional SI, online SI 
models have been shown to have certain advantages over traditional sessions. Painter, Bailey, Gilbert, and 
Prior (2006) note that online SI allows students access to supplemental materials anytime, anywhere. Students 
who are anxious about speaking or solving problems in front of others may find online SI appealing because 


they are not surrounded by other students. The online format allows for easy distribution of additional 


2FITW (CFDA#84.1 1 6F) is " designed to support the development, replication and dissemination of innovative solutions and evi- 
dence for what works in addressing persistent and widespread challenges in postsecondary education for students who are at risk for 
not persisting in and completing postsecondary programs, including, but not limited to, adult learners, working students, part-time 
students, students from low-income backgrounds, students of color, students with disabilities, and first-generation students" (U.S. 
Department of Education, 2015). 
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resources, as well as quizzes and surveys that provide immediate feedback to students. Online SI sessions can 
be recorded and viewed multiple times for students who missed a session or need additional support. Hurley, 
Patterson, and Wilcox (2006) argue that this "time to think" facilitates the deepest level of learning and helps 
students "formulate questions like those modeled by their facilitator, make observations, and consider 
solutions." 

Still, online SI is not without its disadvantages (Painter et al., 2006). The cost of software and related 
equipment, as well as technical support, often makes online SI prohibitively expensive for some support 
programs. Similarly, some software companies set limits on the number of users at one time and therefore the 
number of students that can be served, making their online portals inadequate for conducting SI sessions. 
Moreover, even when the hardware, software, and technical support are suitable, students sometimes lack the 
requisite computer-literacy skills or technology (e.g., microphones and cameras) to fully engage in online SI 
sessions. Finally, in the online environment, SI Leaders may encounter difficulties managing students that 
would not occur in a traditional SI setting. For example, maintaining student attention can be challenging due 
to the students’ ability to easily leave and reenter the discussion . In addition, SI Leaders may encounter 
communication issues with subjects such as math and chemistry because these disciplines utilize unique 
symbols that can be difficult to use in a digital format. 

Although ascertaining the effectiveness of online SI has not received the same level of scholarly 
attention as its traditional SI counterpart, there is some evidence suggesting that participation in online SI 
positively impacts students' academic achievement and persistence. 

Sargent, Borthick, and Lederberg (2013) studied the effect of video-recorded supplementary tutorials 
on student performance in an introductory accounting course. These tutorials were similar to an SI presentation 
in that core concepts were explained and applied to individual problems, strategies and processes for problem 
solving were delineated, and students were encouraged to solve problems for themselves . Each tutorial 
contained informal comments from the facilitator underscoring common mistakes and misperceptions. These 
tutorials were then loaded onto a WebCT platform. The resulting pass rate for low-achieving students who 


accessed the tutorials was significantly higher than the pass rate of the control group, and the tutorial users ' 
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exam scores were increased by a half a letter grade. Sargent et al. (2013) also found that tutorial users 
reenrolled in subsequent semesters at a significantly higher rate than did non-users. Similarly, data from an 
online SI program at the University of Wyoming found that student use of online SI corresponded to improved 
course performance compared to those who did not participate in any SI sessions (Boggs et al., 2011). Finally, 
Ndahi, Charturvedi, Akan, and Pickering (2007) found that a web-based, interactive SI model led to an average 
gain of 14.05% on exam scores in an introductory engineering class. 

Student perceptions of online SI have also been generally positive. Schaffer and Schwebach (2015) 
studied the use of Livescribe pencasts, which are interactive versions of notes and audio, as supplementary 
instruction tools for undergraduate students studying cell biology. Students reported that the pencasts helped 
them learn and retain the material, irrespective of their prior academic performance levels. Use of the pencasts 
increased dramatically before exams, suggesting that students viewed it as an important study tool. Freeman 
and Field (2004) found that more than 90% of industrial technology students in an occupational safety course 
at Iowa State University indicated that online notes and quizzes provided through a WebCT platform helped 
them prepare better for class. Likewise, 89% of students indicated that the WebCT component improved their 
learning experience, and 92% said that they were satisfied with the WebCT experience. Ninety-six percent 
indicated that they preferred to have a class with the WebCT component. 

The focus of the pilot study that TAMUCC conducted from January to May 2015 was a randomized 
control trial to compare the relative effectiveness of online SI with that of traditional SL The goal of the study 
was to collect evidence that might be useful in ascertaining whether online SI might be a viable alternative to 
traditional SI for colleges and universities that wish to enhance the STEM course performance and persistence 
of their students despite experiencing severe resource (e.g., available classrooms) constraints. Thus, the study 
focused on the following research question: do students assigned to online SI perform as well or better than 
students assigned to traditional SI in terms of their persistence and grade earned in the STEM course with 


which the SI is associated? 
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Method 
Participants 

In late January 2015, 644 students in select sections of four STEM courses (Biology 2421, Chemistry 
1412, Engineering 2322, and Mathematics 1442) were randomly assigned to either online SI (the treatment 
group) or traditional SI (the comparison group), with an equal number of students within each course section 
randomly assigned to one group or the other. 

From this initial sample, 17 students were removed because they were enrolled in more than one of the 
four aforementioned STEM courses, 10 students were removed because they dropped their SI-associated 
course prior to the first day of instruction, and 32 students were removed because they opted-out of the SI 
group to which they were assigned. However, as shown in Table 1, these removals did not appear to diminish 
the baseline equivalence of the two groups to any significant degree, as an independent samples, two-tailed, f- 
test of the analytic sample (N =5 85) revealed no statistically or practically significant difference, t(583)=O.10, 
p=0.92, g=0 .00, between the pre-intervention cumulative GPA of the treatment group (ni=277, M;=2.82 , 
SD,=0.63) and the comparison group (nc=308, Mc=2.82, SDc=0.58). Moreover, as shown in Table 2, the groups 
remained equivalent in terms of socioeconomic status, as there was not any statistically or practically 
significant difference in the Pell-grant-eligibility status of the groups, x2(1)=0.66, p=0.42, V=0.03, with 41.2% 
of the treatment group and 44.5% of the comparison group being Pell-eligible. 

In terms of demographic characteristics, 52.0% of analytic sample members were Hispanic; 62.9% 
were female; and 42.9% were low-income individuals (as indicated by Pell-grant eligibility). The results of 
baseline-equivalence analyses for select subgroups in the sample are shown in Tables | and 2. 

Materials 

Members of the treatment group (online SI) and comparison group (traditional SI) were provided the 
same materials throughout the Spring 2015 semester. The only difference between the SI sessions was the 
medium through which the two groups received the materials and interacted with their SI Leaders. Members 
of the comparison group attended SI sessions in a traditional classroom on prescheduled days at prescheduled 


times, occupied the same physical space as their SI Leaders and other session attendees, and received 


25 


Supplemental Instruction Journal, Volume 2, Issue I 


hardcopy versions of all materials. Members of the treatment group also attended SI sessions on prescheduled 
days at prescheduled times, but they did so through an online platform called WebEx video conferencing, 
occupying a completely separate physical space from their SI Leaders and other session attendees and 
receiving electronic versions of all materials. Moreover, online SI sessions were recorded and then available to 
treatment group members who either could not attend at the time of broadcast or who wished to watch the 
sessions multiple times. 

WebEx video conferencing permits users to view multiple video feeds at once or side-by-side with a 
screen sharing mode. During online sessions, SI Leaders can easily redirect questions , facilitate discussions , 
share content, administer quizzes, and track attendance, just as they would in traditional SI sessions. Users can 
share content from their own device, collectively annotate documents, and share ideas on a virtual whiteboard 
from any device with an internet connection. TAMUCC' s use of this platform ensured that the treatment 
group's SI experience closely resembled that of the comparison group while also permitting the treatment 
group to capitalize on online SI's inherent advantages: the ability to receive instruction irrespective of location 
and view recorded sessions anywhere at any time. 

Design and Procedure 

After students were randomly assigned to either the treatment or comparison group in January 2015, an 
equal number of online SI and traditional SI sessions were then conducted weekly (repeated three times on 
different days at different times) through the end of the semester in May 2015. Students in the analytic sample 
had the option of attending as many or as few SI sessions as desired within their group category (i.e., online SI 
or traditional SI). 

SI Leaders were TAMUCC students who had already completed and earned a high grade (B or higher) 
in the course with which their SI sessions were associated. The same SI Leader led both an online SI group 
and a corresponding traditional SI group for each STEM course s/he was assigned. (For example, if SI Leader 
#1 led an online SI group associated with Dr. X's Mathematics 1442 class, SI Leader #1 would also lead a 
traditional SI group associated with that class.) SI Leaders always attended each meeting of the course with 


which their SI was associated and then, in accordance with guidelines established by TAMUCC's SI program 
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manager developed lessons to be delivered in the corresponding SI sessions. 
Results 

The principal goal of this pilot study was to investigate the extent to which online SI might be a viable 
alternative to traditional SL Consequently, the research questions for the study focus on ascertaining whether 
students in online SI perform at least as well as those in traditional SI in the outcome domains of credit 
accumulation (operationally defined herein as persisting in the SI-associated course until the end of the 
semester) and academic achievement (operationally defined herein as grade earned for the semester in the SI- 
associated course). 
Course Persistence 

As shown in Table 3, there was no statistically significant difference between the complete analytic 
sample's treatment (n;=277) and comparison (nc=308) groups in terms of the rates at which they persisted 
(92% for the treatment group vs. 90% for the comparison group) in their STEM courses to the end of the 
semester, x2(1)=0.58, p=0.45, V=0.03. Moreover, as shown in Tables 4 and 5, there were no statistically 
significant differences (1.e., all p>0.05 ) between the treatment and comparison groups when just data from the 
females only, males only, Hispanics only, and Mathematics (MAT) 1442 students only” subsamples were 
analyzed. 
Semester Grade Earned 

As shown in Table 6, an independent samples, two-tailed t-test revealed no statistically significant 
difference, t(583)=0.36, p=0.72, g=-0.02, between the treatment (n,=277, M;=1. 88, SD,;=1.28) and comparison 
(nc=308, Mc=1. 91, SDc=I. 25) groups in the complete analytic sample in terms of semester grade earned’ in the 
STEM courses with which the SI was associated. Additionally, as shown in Tables 7 and 8, no statistically 


significant differences between the treatment and comparison groups were found among the females only, 


3 Subgroup analysis by course was limited to MAT 1442 students because only this course group (N=384) included enough members 
to achieve the statistical power necessary to detect small to medium effects. 


4 Semester grades earned were coded as follows: A=4, 8 =3, C=2, D=1, F=O, and W=O. 
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males only, Hispanics only, and MAT 1442 students only subsamples. 


Discussion 


The results of this pilot study are very promising, as the online SI group appeared to perform as well as 


the traditional SI group in terms of both their persistence to the end of the STEM course and the grades they 
earned for the semester. Moreover, the results were similar across subgroups, suggesting that online SI is a 
tool that can be useful for a broad cross-section of students at institutions of higher education. Consequently, it 
appears that online SI may be a viable replacement (or at least valuable complement) to traditional SI in 
postsecondary settings. However, more research needs to be done to see if these results can be replicated with 
future cohorts, as well as to investigate the extent to which there are differences in performance on more distal 
outcomes (e.g., year-to-year retention, cumulative GPA, graduation, receipt of a STEM degree, and so on) for 
the cohort who participated in SI in Spring 2015. Moreover, the extent to which sample members attend SI 
sessions needs to be examined in future studies and factored into associated analyses so that questions of 
minimal and optimal treatment dosage might be effectively addressed. (In the present study, implementation 
data limitations restricted the analytical focus to intent-to-treat.) Finally, the findings of future studies would 
be enhanced by varying on a weekly basis the order in which SI Leaders utilize specific session formats. For 
example, if a given SI Leader facilitates his or her online SI session first and the corresponding traditional SI 
session second in week one, the SI Leader would do the reverse in week two, facilitating the traditional SI 
session first and the online SI second. The third week, in turn, would repeat the order of the first week; the 
fourth week, the order of the second week; the fifth week, the order of the first week; and so on. 

Future research by TAMUCC will investigate the extent to which these results can be replicated, and 
more distal outcomes might be impacted. Furthermore , TAMUCC will be taking a closer look at the relative 
effectiveness of online SI versus traditional SI in terms of specific STEM disciplines. To facilitate these 
investigations, TAMUCC will be implementing online and traditional SI and collecting data on additional 
cohorts in 2015-16, 2016-17, and 2017-18, as part of a comprehensive program of resources to produce 


definitive and consequential evidence of online SI's utility and effectiveness. 
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Table 1 
Baseline equivalence of academic achievement* for analytic sample 


Group N ny M, SD; Ne M, SD. t df p &§ 


All sample members 585 | 277 2.82 0.63 308 2.82 0.58 | 010 583 0.92 0.00 


Females only 368 | 168 2.88 0.63 200 2.80 0.60 | 1.10 366 0.27 0.13 
Males only 217 | 109 2.75 0.62 108 2.85 0.54 | 1.29 215 0.20 | -0.17 
Hispanics only 304 | 142 2.73 0.61 162 2.77 0.58 | 0.70 302 0.49 | -0.07 


Mathematics (MAT) 
1442 students only 


*As measured by pre-intervention, cumulative GPA 


384 | 185 2.74 0.61 199 2.74 0.60 | 0.07 382 0.94 0.00 


29 


30 


Table 2 


Baseline equivalence of student socioeconomic status * for analytic sample 
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Group N | 


All sample members 585 


Females only 368 
Males only 207 
Hispanics only 304 
MAT 1442 only 384 


A 


277 
168 
109 
142 
185 


nt(Pell-eligible 


114 
80 
34 
69 
89 


*As measured by Pell-grant eligibility status 


Table 3 


STEM course persistence_of analytic sample 


Group N 


n 


n,(Persisting) 


All sample members | 585 


Table 4 


Zid 


255 


Ne 


N,(Pell-eligible) 


137 
101 
36 
88 
95 


STEM course persistence by select subgroups (Females only, Males only, and Hispanics only) of analytic 


sample _in STEM courses 


Group N 
Females only 368 
Males only 217 
Hispanics only 304 


ni Persisting) 


155 
100 
132 


n.(Persistin, 
Bg 


182 
96 
144 


Wald* 


0.10 
0.69 
1.56 


of 
1 0.75 
1 0.41 
1 0.21 


VOR 
0.03 0.87 
0.03 0.87 
0.02 0.91 
0.06 0.79 
0.00 1.01 
Vv OR 

0.03 1.25 
Exp(B)* 
0.47 
0.68 
0.60 


*Treatment (i.e., online SI vs. traditional SI) variable results from a binary logistic regression utilized to 


provide statistical control because baseline equivalence in cumulative GPA (cf., Table 1) is insufficient (i. e., 


effect size difference between groups at baseline is not :S_ 0.05) 
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Table 5 


STEM course persistence by select subgroups (WA TH 1442 students only) of analytic sample in STEM 


Courses 

Group N nt n,(Persisting Ne n,(Persisting x2 df p ¥ OR 
MAT 1442 only 384 | 185 173 199 179 159 1 0.21 | 0.06 1.61 
Table 6 


Semester_grades earned by overall analytic sample in STEM courses 


Group Ni[m M SD| % M SDz t af p & 
All sample members | 585 | 277 =1.88 1.28 | 308 1.91 1.25 | 0.36 583 0.72 | -0.02 
Table 7 


Semester grades earned by select subgroups (Females only. Males only, and Hispanics only) of analytic 


sample _in STEM courses 


*Means adjusted via ANCOVA procedure with pre-treatment cumulative GPA as covariate. This ANCOV A 


M. SD, Mxe| EF df p |g 


Group Nin Mm SD Me 


Females only 368 | 168 1.94 1.22 1.90 | 200 1.97 1.25 2.01 | 0.99 1 0.32 | -0.08 


Males only 217 | 109 1.80 1.36 1.84 | 108 1.81 1.25 1.77 | 0.19 1 0.66 | 0.05 


Hispanics only | 304 | 142 1.68 1.25 1.71 {| 162 1.88 1.20 1.86 | 1.40 1 0.24 | -0.12 


procedure was selected because baseline equivalence in pre-treatment cumulative GPA (cf., Table 1) was 
initially insufficient for these subgroups. 
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Table 8 


Semester grades earned _ by select subgroups (MATH 1442 students only) of analytic sample in STEM courses 


Group N | M, SD, Ne mc SDe t af fe & 


MAT 1442 only 384 | 185 1.80 1.29 | 199 1.81 1.28 fon 382 0.92 | -0.01 


